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ABSTRACT 

The two subjects of this paper are; (1) the 
development of the training consortia concept, and (2) the process 
employed during three visits to evaluate each of three training 
consortia now in operation. The training consortia have as their goal 
-^0 training of personnel to work as members of collaborative teams 
in the areas of educational development, diffusion, and evaluation. A 
copy of the original Request for Proposal is provided and referred to 
throughout the paper. It is concluded that a systematic strategy for 
evaluating RDD and E training programs has not yet been installed. 
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Do We Practice What We Preach? Are Systematic Research, 
Development, Diffusion, and Evaluation Being Used in 
Establishing RDD&E Training Programs? 

The Evaluation of Educational RDD&E Training Programs, by 
John E. Hopkins. 
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In this paper I will comment on two matters: (1) the development in reality 



j 

of the training consortia concept, which is one segment of the activities being 



supported by the Research Training Program of the National Center for Educational 



Research and Development, U.S. Office of Education, and (2) the process employed 
during three visits to evaluate each of the training consortia now in operation. 
I was a member of each of the three evaluation teams which site visited these 
projects in late October and early November, 1971. My comments in this area are 
based entirely on my own observations as a member of these site visit teams and 



have no substantiation beyond that. 

Perhaps it is needless to say that I will not comment on the worth or value 
of any of the three projects individually, for reasons which are obvious. Neither 
will I comment on the progress of the three projects generally, for reasons which 
I shall itemize later. To repeat, the focus of this paper is on the actualization 
of the concept, and, secondly, the process employed to evaluate the new RDD&E 



training consortia . 

to 

» 

(^Actualizatio n of the Training Consortia Concept: s 

la March, 1970, the Research Training Program issued a Request for Proposals 
^(RFP 70-12) to design new patterns for training educational RDD&E personnel. Twelve 
^groups were funded to conduct design studies during the six-month period June to 
(J^jjDecember, 1970. In January, 1971, three of the twelve were funded to conduct a 
.. three -year operational test of their designs. Upon evidence being obtained of their 
^effectiveness, federal support of their activities was to be continued, being phased 
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( out gradually over an additional seven year period. Thus, a federal commitment 

(to the extent federal agencies are able to make a "commitment”) was contemplated 

' for a full ten year period, assuming successful completion of the initial three- 

year operational test phase. The training consortia at Ohio State, under the 

direction of Dan Stufflebeam, at LRDC in Pittsburgh, under Glen Heathers and 

« 

John Yeager, and at the Far West Lat oratory for Educational Research and Development, 

c 

v 

under Paul Hood and Joe Ward, recently completed the first year of that three-year 
operational test phase. 

At this point I want to give you some background as to what these consortia 
were intended to be. On the first page of the supplementary materials which were 
made available to you, you will see a listing of three general objectives which 
appeared in the original description of the training consortia concept (RFP 70-12). 
Below the ^general objectives are eight speci^ .c objectives which relate to the first 
two general objectives. As you look over those general and specific objectives, I 
believe you will get the picture of a group of training structures which were. rich 
in hitman and financial resources; powerful enough to create new and imaginative 
approaches to a kind of training that is not very widely offered in the United States 
today, that is, the training of personnel to work as members of collaborative teams 
in the areas of educational development, diffuu^on, and evaluation; structures which 
would produce field-oriented people to work on the basic and important problems 
which confront public and private schools at all levels- -today And tommorrow. 

This concept ia not now receiving a true test. Under the funding conditions 

imposed upon "the training consortia by the Research Training Program, and the 

* * •• 

« 

budgetary restraints imposed on the Research Training Program by Congress, the 
training consortia are a long, long way from the resource-rich environment in 
which creative change flourishes, and which, in turn, helps to make organizational 
change possible. Because of the lack of funds, the three training consortia are 
receiving between half to two-thirds of the funds originally thought necessary to 
^ nV/ .mpleme n t the concept. 2 
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Under these circumstances, the training consortia are struggling in the 
directions laid out in the objectives, but can hardly be said to be moving 
with strength or power. Under present funding restrictions, there is no 
possibility of their exploring multiple alternatives to known methods of training. 
With funds as tight as they are, the consortia will not be able to do as much 
developmental work as had been hoped, and it appears unlikely that they will be 
able to develop the cost^accounting and phenomenological data bases which were 

i 

considered necessary for evaluating the relative effectiveness and efficiency of 
their individual operations. 

.From an outsider's point of view, at least, it seems that, should their .present 
anemic condition continue, the products of the three existing consortia can only be 
pale shadows of the robust forces for change and improvement which were originally 
envisioned. However, there is still ample opportunity for these fledgling units to 
grow into^ the roles expected of them, I would strongly recommend that consideration 
be given to transfusing additional funds into at least some of the consortia -- so 

i 

i 

they have a real chance to prove themselves. I would not be optimistic about what 
they could prove, or we could learn, if the present situation were to continue 
without substantial change. 

In this symposium we are looking at the question of whether we are practicing 

* 

what we preach. I submit that the way of research is to develop and assess the 

* 

critical test, not simply to carry on as best you can under the circumstances. If 
we were to apply the principles of research to’ this matter, then, I believe we would 
do that which was necessary to secure a full test of the concept. By so behaving, 

it is at least possible that we could learn some things which would help us to chart 

% • * 

our goals and means for the future. 




Process Employed to Evaluate the Training Consortia 

Each of the three evaluation teams carried out what I believe was a quite 

conventional site visit. By that I mean that each member of each team received 



the materials carefully. Each of the visits to the site was occupied principally 

by a day to day-and-one-half re-examination and/or elaboration of these same 

/ 

m a terials /by members of the project staff. Some tine was devoted in each visit to 

short interviews of staff and trainees by the site visitors. On the basis of 
I 

essentially impressionistic data* the members of the visiting team then formed 
preliminary individual judgments about the 'project generally and/or about specific 
components of the project. These judgments were subsequently pooled and modified 
through discussion among members of the evaluation team. Finally, a consensus set 
of conclusions was developed and delivered to the project staff. 

The results of the three visits are informative. At one of the sites, the 
evaluation team, in effect, quarreled with the project staff with respect to the 
objectives which they had developed and operationalized for their local training 

t * 

effort. The members of the visiting team reached the consensus conclusion that there 
were more desirable sets of objectives toward which the consortium could be working. 
The result of this visit was that the project staff was required to recast their 
program substantially. 

At a second site, the evaluation team questioned the means being developed by 
the project staff to meet their objectives -- which, in this case, the evaluation 
team approved. The result of the visit was that the project staff was required to 
reformulate the means being developed to provide the training being offered in their 

particular program. 

At the third site, the evaluation team had no argument with either the 
objectives or the means proposed by the project staff, and so neither recasting 
^or reformulation were required of these trainers. The evaluation team .ended this 
visit by offering a pat on the back and commendation for a job well done. 

At all three sites the members of the visiting teams were business-like, hard 
. working, alert, sensitive, forthright, and courteous. That is to say, in terms of 
a conventional site visit, I suspect most participants believed we had acted as 

a good and effective evaluation team. 

o 

ERIC 



4 



Before I comment further on the evaluation process used, I think it only fair 

to provide you a number of reasons why this fairly conventional approach to 

evaluation team- was developed and used, the main reason being, as you will 

see, that it would have been extremely difficult to mount and conduct any 

other kind of evaluation. * • 

• * 

1. The evaluatior teams were visiting the three training consortia very 
early in their organizational lives. This was the first annual visit 
during a three-year operational test phase for projects which have the 
potential to continue under federal funding for up to ten years. 

2. The major emphasis of the training staffs during this first year was 
on the organizational and operational, rather than the evaluative. 

Their first responsibility was to organize themselves, recruit and 
select a staff and class of trainees, and begin offering a training 
program within 4-7 months. . 

3. The operational test phase was explicity stated to be a developmental 
period, during which experimentation and flexibility were encouraged. 

The situation at each location therefore remains fluid, which tends to 
inhibit the development of data- gathering and analysis procedures, 
instruments, and techniques. 

4. Since all of the projects are seriously underfunded, the resources at 
their command are strained simply to cover the essentials necessary to 
offer a program of training. 

5. All of the consortia had heavy reporting requirements during the year, 
which imposed a further drain on their already- slim human resources. 

6. None of the programs had any "graduates,” so It would not have been 
possible to make any assessment of the products of the programs — even if 
any of them had spelled out the long-term competencies they were trying to 
Impart, which none of them had. 

All of the above are reasons why the site visit teams adopted the evaluation 
mode actually used. Operational .realities seemed to compel use of the conventional 
site visit approach 1 I don’t wish to say the first evaluations should not have 
been carried out in that manner. X do wish to say that it would be distinctly 
undesirable for the annual evaluations to continue in this mode. Continuation of 
the conventional site visit approach over the long term would be both inappropriate 
and damaging to the training consortia program. "Inappropriate" because the 
conventional site visit is not suited to the type of performance- based program which 
is specified for the consortia. A performance-based program should not be evaluated 
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on the basis of impressionistic data. ’’Damaging" because the project staffs will 

I 

X 

be unable to progress if they continue for any length of time in a situation where 

/ 

even fundamentals may be called up for question, e.g., fundamentals such as the objectives of 
the project' or the means developed to achieve them. 

If you will refer again to the supplementary materials, at the top of page 2 
(Section A) you will find the evaluation plan contained in the original RFP. Note, 
especially, that evaluations of the consortia were to be conducted **. . .‘ on the 
basis of short- and long-term performance and program criteria specified by the 
training staff of the consortium and agreed to by the Research Training Program." 
Elsewhere in the RFP it is explained that "short-term" refers to the development 
of specified skills, behaviors, and understandings, while "long-term" refers to 
such matters as the placement and subsequent productivity of graduates, program 
objectives to be accomplished, and so forth. Originally it was anticipated that 
each consortium would develop and maintain a data-gathering mechanism which would 

periodically provide data in these areas, for administrative, instructional, and 

\ 

evaluational purposes. ■* 

During the October - November site visits, none of the consortia was able to 
provide either short- or long-term performance and program criteria — approved or 
unapproved by the staff of the Research Training Program. No data were yet 
developed on the progress or productivity of the trainees. Consequently we looked 
as broadly as we wished, ending up with the objectives in one project and the means 

in another. 

Bad Jdie consortia been able to -develop performance and program criteria and 
. . # *. 
gotten them approved by the staff of the Research Training Program, there would 

have been two beneficial results. First, the RTP staff would have seen in performance 

terms the implications of operationalizing given sets of objectives, and would have 

been able to negotiate at the outset a mutually acceptable set of objectives. The 



project staff would then have had a firm, approved base from which to proceed. 

0 :ond, the matter of means would have been removed from site /isit consideration, 



because, as in performance contracting anywhere, the contractor would have been 
freed to use whatever honorable means he chose to produce the results he promised 
within the funds granted, of course - 

On the third project we commended the project staff, but we still don Vt. JSS £2 
whether the program they are developing is going .to be effective, or efficient, or 
viable in dollar terms in preparing the kind of people they are trying to produce. 



Until the data-gathering mechanisms and procedures originally called for are 

1 * 

developed, there will be no way available to make these judgments. 

Obviously I believe that the process of evaluation conducted with the consortia 
should be cf the type outlined originally. If you will refer to your supplementary 
materials one final time. Section B, which begins on page 2 and continues through 
pages 3 and 4, contains the areas examined when the twelve design proposals were 
being evaluated. The categories of (1) students, (2) faculty, (3) programs, (4) 
participating institutions and agencies, (5) evaluation capability, (6) administration, 
and (7) projected output in relation to cost appear to me to have utility for 
evaluation of the consortia, as well. The sub- items under each category heading 
offer a minimal structure for developing the performance and program criteria 
specified in the original RFP. 

In conclusion, the only answer I can provide to the second question posed in 
the title of this symposium is: "No, a systematic strategy has not yet been 

installed for evaluating BDDE training programs." An evaluation strategy exists, 

I think. Operational realities have so far prevented the conceptual niceties from 
* being implemented, but I believe there is a possibility that the needed summary 
quantitative data may, indeed, be available for examination a year or two from now. 

I also believe that if this is going to be the case, it will be necessary for the 
project directors and the staff of the Research Training Program to install the 
development of performance and program criteria as a primary work objective for the 

coming year. Possibly then we will be able to report that we do practice what we 
preach, because we evaluate in terms of previously specified and mutually agreeable 
ERJC performance and program criteria. 



Request for Proposals (No. 70-12) to Design New Patterns for Training Research, 

Development, Demonstration/Dissemination, 
and Evaluation Personnel in Education 

Overall or General Objectives 

1. Secure greater balance in the type of persons being trained by developing training 

programs which focus on educational D,D, & E to complement the existing training 
programs in educational research. ^ 

2. Produce persons trained tcxworfc collaboratively with professional, technical, and 

para- professional associates, to balance the number of solitary, part-time investi- 
gators being produced by existing training programs. * 

3. Provide on-the-job training and/or retraining. 

Specific Objectives Relating to General Objectives 1 and 2 

1. Secure tested patterns for training persons to engage in collaborative endeavor, as 
members of R&D teams. 

2. Encourage (a) experimentation with apprenticeship/internship arrangements, and (b) 

course content development activity. 

\ • 

3. Produce new knowledge about training, and possibly influence favorably the training 

models now commonly in use. \ • * 

4. Stimulate challenges to current assumptions about training so as to encourage funda- 
mental reconceptualization. 

5. Involve user institutions in training in order to enhance the possibility of new 
'approaches to training being developed. 

6. Mass resources, rather than spread them, to secure synergistic benefits such as (a) 
the building of a ’’critical mass" of conceptual talent, (b) opportunities for 
specialization among the participating trainers, (c) development of a power base of 
sufficient strength to effect changes in established institutional requirements and 
procedures, and (d) certain economies of scale. 

7o Promote greater understanding among trainees of the contributions made by the various 
specializations. 

Secure better qualitative and greater quantitative input from the disciplines and 
professions related to education. 
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The training programs will be evaluated annually for three years on the basis of 
short- and long-term performance and program criteria specified by the training 
staff of the consortium and agreed to by the RTP. Each consortium will be expected 
to develop and maintain a mechanism for securing periodic data on the progress and 
• productivity of its trainees and "graduates." The evaluation will be conducted by a 
team of 2-3 on site visitors who will report their observations and conclusions to 
the RTP. A judgment as to the viability and relative success of the training 
patterns will be made toward the close or at the end of the third year of operation. 

B. Evaluation of Contractual Effort — pp. 20-3. RFP 70-12.~. 



The design for training consortia will be evaluated on the basis of the quality, 
adequacy, innovativeness, and/or feasibility of the proposals made with respect to 
a> students, (2) faculty, (3) program, (4) participating institutions and agencies, 

(5) evaluation, (6) administration, and (7) projected output (of personnel and materials) 
in relation to overall costs. Specifically, the evaluation will focus upon such factors 
as: 



(1) Students 



a. Plan for recruitment over a national or broad geographic region; from 
minority groups,; from related disciplines and professions. 

: b. Procedure for assessing level of student talent and other student 

characteristics; criteria to be. used for acceptance into the program. 

c. Plan for assessing student knowledge, diagnosing student's needs in 

relation to his personal objectives and the program's objectives, and 
determining progress periodically. - 

d. Procedure for assuring student participation with faculty and students of 
other training programs in important investigations and/or problem- 
solving projects and programs. 

e. Plan for actively aiding trainees to find employment where they can use 
their newly-acquired skills once they have completed the program. 

(2) Faculty specifically attached to the training program 

a. Participation of faculty of superior quality on a full-time or near full- 
time basis. 

b. Current or recent involvement of faculty in research, development, 
demonstration/dissemination, or evaluation programs or projects. 

c. . Procedure for regular faculty consideration of the status, progress, and 

objective of the training /development being carried out, and provision 
for systematic efforts by the faculty to improve them. 

d. Contribution to the training program by faculty from related disciplines 
and professional fields. 



(3) Program 

a. Objectives coincide with or are responsive to needs identified by 
extant manpower studies and/or stated national priorities. 

^ • i 

b. Training is directed toward solution of significant problems in 
American education generally. 

c. Content has been organized, developed, or tailored to meet the needs 
of the trainees, i.e., is more than the regular on-going program of 
the institution (s) . 

d. Abundant apprenticeship opportunities are provided. 

e. Consideration has been given to modifying standard instructional modes, 
patterns, time periods, content, and locations for the purpose of in- 
creasing the effectiveness and efficiency of the training offered; 
imaginative manipulation of these elements is desired. 

f. Needed facilities are available for use. 

g. Abundant research, development, demonstration/dissemination, and/or 
evaluation projects and programs are being carried out nearby and are 
open to observation and participation. 

h. Consideration has been given to providing a supervised internship in 
settings where the trainee might reasonably expect to be employed upon 
completion of his training program. 

i. Students in similar or previous programs offered by the constituent 
units completed the entire training program and, where offered, the 
appropriate degree program. 

j. Provision for interaction between trainees working on different problem 
areas, functions, or levels of competency. 

k. Willingness and ability to mount and carry oui course content and/or 
instructional materials development activities is demonstrated. 

l. Program is sufficiently rationalized to be adapted to a different but 
similar set of institutions and agencies. 

* • / 

(4) Participating institutions and agencies 

« 

a. " Adequacy of the physical and financial resources for training which 

are available. 

* •* * * 

b. Evidence of ability to attract and retain faculty and students of 
high quality. 

c. Ability of training faculty to conceptualize .id put into operation new 
training programs for new education roles. 

* . 

d. Evidence of efforts to innovate or substantially improve education. 

a. Existence of other training programs of superior quality. 



f. Evidence of ability to secure projects and programs to serve as 
apprenticeship sites for trainees. 

(5) Evaluation 

a. Objectives specified for both the short-term (i.e., experiences and 
performance of skills) and long-term (i.e., placement and productivity 
of "graduates") • 

b. Plan for monitoring training to secure periodic feedback on progress • 

vhile training is in process; procedures for feeding monitoring data 
into the decision-making process. * 



c. Plan for regular reporting of summary quantitative data on the 
characteristics, progress, and placement of trainees. 

d. Procedures for securing cost data in relation to products of the 
programs being offered. 

(6) Administration 

a. Identity of the director of the consortium. 

|), prime responsibility for operation of the consortium designated. 



c, plan for operation of the consortium on the basis of continuous data 
| input from evaluation component. 

Plan for periodic conceptual and design inputs from sources external 
to the consortium. 

(7) Projected output in relation to costs 

a. projection of number and type of trainees. 

b. Capacity for expanding the training offered. 

c. Production of new course content or instructional aids in tested, 
generalizable form. 

’ d. Plan for a cost accounting system which will provide accurate summary 
data on experimental programs and activities. 




